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ABSTRACT 



This document contains three papers on issues in evaluation. 
"Evaluation of the Method of Modeling: A Case Study of the Finnish Steel 
Industry" (Ville Nurmi) describes the method of modeling as an educational 
strategy to support both specific goal-directed transformative learning 
focused on work process and learning in workplaces, and it evaluates the 
effectiveness of modeling as an educational strategy in an industrial 
setting. "Performance-Level Evaluation Methods Used in Management Development 
Studies from 1986-2000" (Doris B. Collins) reports on a study in which Burke 
and Day's 1986 meta-analysis of the effectiveness of managerial training and 
the Results Assessment System lens are used to analyze 18 management 
development studies from 1986-2000 that had performance- level outcomes. 
"Redefining Performance: Productivity and Return on Investment in Physical 
Therapy" (Scott S. Harp) is a theoretical paper designed to begin the 
conceptual analysis of the feasibility of developing and using a tool 
measuring return on investment (ROI) to measure ROI in training in the 
service-oriented field of physical therapy. Two papers include substantial 
bibliographies. (MN) 
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Evaluation of the Method of Modelling: A Case Study of the Finnish Steel Industry 



Vi lie Nurmi 

Rautaruukki OYJ , Metform 



The study aims to evaluate the method of modelling. The method seeks to improve professional 
knowledge and skills of the participating persons. It also aims to develop the work processes , 
which are modeled. The method has been developed for the purposes of the pulp and paper 
industry to facilitate the transition from the manual process control to the automation systems. 

This case study is the first application in the steel industry. 

Key words: Evaluation, Industry, Modelling 

The paper has two intentions. First, to described the method of modelling as an educational implication to support 
both specific goal-directed transformative learning focused on work process, and learning in specific contexts, i.e. 
work places. Second, to evaluate the application of the method in order to identify its effectiveness in the industrial 
setting. 

Problem Statement and Theoretical Framework 

The method of modelling has been invented and developed by the researchers of the Finnish Institute of 
Occupational Health for the purposes of pulp and paper industry, during the transition from the manual process 
control systems to the automation systems in the late 1980s and early 1990s. In this study, the method is applied in 
the steel industry for the first time. Even though, the basic structure of the metal industry is quite the same than in 
the pulp and paper sector, it is necessary to identify the effectiveness of the method in this area. The modelling has 
been implemented also in the health and service sectors. There are not obvious limitations, why it could not be 
applied in almost any type of industry, where exists a need to improve the work process and vocational knowledge 
and skills of the personnel. 

Individuals construct the various communication situations and countless observations continuously. These 
observations become mental models by the thinking process which human beings implement consciously and 
unconsciously. There are also mental models about the work processes. In order to improve the knowledge and skills 
of the personnel the mental models have to be shared and formulated to correspond the reality of the work place. 
The method of modelling of the work process has been developed to advance sharing of the individually constructed 
mental models among the members of the work unit. The transformation from the manual control systems to the 
automation systems was the practical need for the method (Bainbridge, 1992; Norman, 1983). 

Research Questions 

As mentioned in the problem statement section, the paper aims to describe the method of modelling and to evaluate 
it. There are not research questions for the description part. But it is evident that the description is necessary in 
building the foundation for the evaluation of the method. 

For the evaluation part of the study, there are three specified research questions. First, is the method of 
modelling effective in improving the conceptual mastery of work among the participants? Second, is the method of 
modelling effective in improving the work atmosphere among the participating personnel groups? Third, is the 
method effective in improving the productivity of the production unit in the target organization? ' 

Methodology 

The description part of study includes a plain clarification about the method. It builds a foundation for the second 

Copyright © 2001 Ville Nurmi 
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empirical part emphasizing the evaluation of modelling. The evaluation part includes three methods referring to the 
research questions. 

To answer the first research question, the test of conceptual mastery of work is applied. The participating 
workers of the target organization answer the paper-and-pencil questionnaire containing basic conceptions of their 
everyday working life. Study subjects are asked to mark for each proposition, whether it is correct or false. As a 
result, the test provides a percentage number referring to the share which a person masters of the all crucial concepts 
of his work. The test is done before and after the modelling is employed. Besides the participating production line of 
the steel factory, the personnel of the four other production lines answer the test, too, to provide control group 
setting for the study. The changes in the conceptual mastery between the pre and posttests among five production 
lines are analyzed through ANOVA procedures (Leppanen, 1993; Nurmi, 1998, 1999). 

To answer the second research question, a work atmosphere assessment is applied in the participating 
production line, as in four other personnel groups. The measurement takes place after the implementation of 
modelling. The survey tool is "Healthy Organization", which is developed for the purposes of working life in the 
European Union. Finnish expert organization disseminating this tool is the Finnish Institute of Occupational Health. 
Differences between five production lines are analyzed through ANOVA procedures. 

To answer the third research question, the production figures, the amount of occupational hazards and 
absenteeism amounts of the five production lines are compared and analyzed through statistical procedures. 

The first two analyses are implemented till the end of year 2000. The data for the third section could not be 
gathered before the end of the year 2001. Because the adequate follow-up time period must be at least one year in 
the large, slowly changing organizations, such as is the steel factory. There are five production lines in the factory. 
Each line has approximately 20 workers. Therefore, the respondent group in all is about 100 persons. 

Results 

Description of the Method of Modelling 

The modelling refers to the systematic analysis of the work process. The participators, who are the 
operators and supervisors themselves, address each major attribute of the work. It includes also theoretical analysis; 
why certain processes are done as they currently are. The ultimate goal is to develop simultaneously the work 
process and vocational knowledge and skills of the operators (Leppanen, 1993). 

The analyzed elements are the following: 

• the goals of the work (what are we intending to accomplish) 

• the target of the work (what are we changing, working up) 

• the equipment of the work (by which tools are we working up) 

• the worker (who does the work) 

The mental models and the effective usage of them are the focus points. The modelling aims to expand the 
mental models and understanding of the work process, and to make the knowledge transfer from the expert workers 
to the newcomers more effective. The modelling is entirely based on the expertise of the participants, and it is fully 
focused on the actual work process (Leppanen, 1993; Leppanen et al., 1996). 

In the modelling, the participants analyze the work process in terms of products, raw and additional 
materials, machines and processes, co-operative networks, and critical process phases. Each area is modelled further 
according to the framework created by the participants. For instance, the product model may contain the following 
dimensions: 

1. List of the products and clients, 

2. the purpose of use, 

3. special requirements, e.g. quality, 

4. exceptions in manufacturing, 

5. possible problems in manufacturing, 

6. the percentage share of production, and 

7. The development needs of the product. 

The models are analyzed and progressed in the meetings where representatives of several modelling teams 
are working together. The external consultant or the supervisor of the participants facilitates the modelling. 
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Contributions to HRD and Future Challenges 



There are several challenges for the method. The roots of the modelling are in the heavy industry, but it can be 
applied to the various areas of human activity. It has been used already in the service sector, e.g. communal 
canteens, and in the provisions industry. The latest experiment in one training course was to model the various work 
processes, products, and raw materials of a professional ice-hockey team. 

The challenge is to develop the internal organizational players to be ftcilitators of learning. The process 
may be more successful, if the external advisors coordinate the application, but the continuous improvement will not 
be achieved through this way. The supervisors and other foremen have to face the fact that team-building decreases 
the needs for the traditional management actions, such as control and decision-making. The modelling is a tool, 
which change agents can apply in developing the process with the personnel. 

The models include development ideas for the products, the processes etc. It takes a tremendous amount of 
efforts to implement or even just process all of them. For the participants, it is necessary that they see their own 
chances to influence. They can assimilate the development of the work to be an evident element of their work 
practice. It would increase the effective iBage of the knowledge capacity of the personnel, if the operators 
continuously plan, how the process can be improved to be more appropriate. 

Undoubtedly, there are several ways to develop the method to better meet the various needs of the work 
place learners and to promote on-the-job learning. Although, the main intention of the method is not in the 
emancipation or enlightenment of the workers, it includes elements that can be applied in other types of 
transformative and action learning activities and programs, as well. 
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Performance-Level Evaluation Methods Used in Management Development Studies from 
1986-2000 



Doris B. Collins 
Louisiana State University 



This research begins with Burke and Day's 1986 meta-analysis on the effectiveness of managerial 
training as its starting point, and uses a Results Assessment System lens to analyze 18 
management development studies from 1986-2000 that had performance-level outcomes . Studies 
included involved managers, leaders, and/or executives. This research was completed to develop 
an understanding of common evaluation characteristics of performance-level evaluations because 
the number of management development programs is expected to grow in the next decade. 

Keywords: Evaluation; Management Development; Organizational Performance 

Evaluation of management development programs is not a new phenomenon. Kirkpatrick’s model for evaluation of 
training and development has been used for forty years to measure training effectiveness. Yet, research indicates that 
organizations are spending little time evaluating the effectiveness of their management development programs 
(Sogunro, 1997). It appears that many corporations take for granted that management development efforts will result 
in improved management skills without valid data to prove the return-on-investment (ROI). 

Annual budgets for management development programs are expected to grow throughout the next decade 
as companies “recognize the shortage of talented managers, the importance of developing bench strength, and the 
need to widen perspectives in order to compete globally” (Gibler, Carter, & Goldsmith, 2000, p. xii). Organizations 
are concerned about the management inadequacies of their employees and are committing to education and training 
that deepen the skills, perspectives, and competencies of their managers. A review of evaluation literature shows that 
there has been a resurgence of interest in the evaluation of management development programs by HRD 
professionals (Moller & Mallin, 1996). However, questions remain as to whether existing evaluation models 
adequately measure the intended outcomes, especially where the goals of management development programs are to 
enhance organizational performance. 

Management development is defined as every form of growth or stage of development in the life cycle that 
promotes, encourages, and assists the expansion of knowledge and expertise required to optimize one’s management 
potential and performance (Brungardt, 1996). Training evaluation is defined as the systematic collection of data 
regarding the success of training programs (Goldstein, 1986). Training evaluation occurs when specified outcome 
measures are conceptually related to intended learning objectives (Kraiger, Ford, & Salas, 1993). Evaluation is 
normally conducted to determine if training objectives were achieved or the accomplishment of training objectives 
resulted in enhanced performance of the individual on the job. It enables trainers to continuously monitor their 
programs and to identify points of intervention for program improvement. In addition, evaluation is becoming more 
critical as human resource departments are being asked to justify their training programs and compete for limited 
financial resources. 

In an unpublished study by Collins (2000), only 16 out of 54 management development studies from 1986- 
2000 had organization-level performance (Rummler & Brache, 1995) as the outcome variable. This leads us to 
wonder why so few studies measure the organizational effectiveness of management development programs. This 
research analyzes management development interventions with a Results Assessment System lens (Swanson & 
Holton, 1999) to determine common characteristics among studies evaluating organizational performance 
improvement efforts. It is done in an effort to provide some theoretical undergirdings for future research to 
determine underlying reasons organizations are spending so little time evaluating the effectiveness of their 
management development programs. Hopefully the findings will influence HRD professionals or corporate 
managers to measure performance at the organization level (Rummler & Brache, 1995), or inspire organizations to 
evaluate programs when they might not have a tendency to do so. 



Copyright © 2001 Doris B. Collins 
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Using the Results Assessment System as a HR Tool 



The Results Assessment System (Swanson & Holton, 1999) defines outcomes broadly to relate to the results of 
changes in leadership style in top management, worker satisfaction, work teams, or organizational change (Lynham 
& Swanson, 1997). The Results Assessment System provides three outcomes levels, each subdivided into two 
outcome categories. Performance level outcomes are subdivided into system or financial and learning level 
outcomes are subdivided into expertise or knowledge. System outcomes are defined as mission-related outputs in 
form of goods/services, having value to the customer, that are related to the core organizational, work, process, and 
group or individual contributor to the organization. Financial outcomes measure the conversion of outputs of 
goods/services attributable to the intervention to money and financial interpretation. Expertise outcomes measure 
whether human behaviors have effective results and optimal efficiency, acquired through study and experience 
within a specialized domain. Knowledge outcomes are defined as mental achievement acquired through study and 
experience. Outcomes with a performance-level result are the focus of this study. Perception level outcomes are not 
a focus of this research. 

The assessment of organizational performance requires that mission-related performance outcomes be 
carefully specified and connected to the mission of the system (Swanson & Holton, 1999). Performance-level 
outcomes can be at the whole system level (organization), a work process within the system (subsystem), or a work 
group (team). According to Swanson and Holton (1999), “every intervention should lead to systems outcome(s) at 
some point” (p. 69). Outcomes are defined as “measures of effectiveness or efficiency relative to core outputs of the 
system, subsystem, process, or individual” (Swanson & Holton, 1999, p. 69). 

Methods 

A qualitative research method was used to analyze management development studies from 1986 - 2000 that had 
performance-level outcomes and involved managers, leaders, and/or executives. Studies that were included were 
located by conducting computer searches with UNCOVER and AB1 Inform, scanning bibliographies, reviewing 
journal articles and proceedings from the annual conferences for the Society of Organizational and Industrial 
Psychology and the Academy of Human Resource Development. Also included were the applicable management 
development studies cited in The Impact of Leadership by Clark, Clark, and Campbell (1992). Key words used in 
the computer search include evaluation, assessment, outcomes, impact, effectiveness, and influence in combination 
with the following subject areas: leadership development, managerial training, management training, management 
development, executive development, leadership education, leadership, management education, and management 
skills. This research located 18 management development studies with an organizational impact since Burke and 
Day’s (1986) meta-analysis of the effectiveness of managerial training. 

Once studies were located and reviewed, codes were provided for the category of management 
development intervention, the training content area, and the intended outcome of the management development 
experience. The management level of the individuals in the sample, the size of the sample, the type of management 
development intervention, and the instrument(s) used to measure outcomes were also recorded. To validate the 
coding scheme, each article was reviewed a second time and the coding choice was compared with the first review 
to ensure similarity. In addition, two doctoral-level human resource development students read four of the most 
recent studies and independently coded the sample studies. One rater disagreed in only one coding element, but the 
raters discussed and agreed on the appropriate coding. 

Intervention Categories 

Leadership development studies include not only formal training, but also a full range of experiences that 
are suggested by McCauley, Moxley, and Van Velsor (1998) to include mentoring, job assignments, feedback 
systems, on-the-job experiences, developmental relationships which include exposure to senior executives, and 
leader-follower relationships. Experiences found in the 18 studies of this analysis fit into intervention four 
categories: 

• Developmental Relationships: Experiences in work settings where another individual influences the 

manager’s personal development such as one-on-one mentoring or coaching. 

o Formal Training Programs: Structured training programs designed to develop the individual manager. 

• Job Assignment: Assignments to an entire job, as redesigning a system or part of a job, or serving on 
temporary task forces. 
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• Structured Experiences: Group activities that include goal-directed, live-action, and task-based interactions 

such as leaderless group discussions, simulations, and targeted exercises. 

Training Content Areas 

Competency areas featured in the high-performance leadership competency model (Holton & Naquin, 
2000) provided a basis for categorizing studies by training content areas. The high-performance leadership 
competency model was used as it provides the only known definition of management development outcomes that 
includes “improving performance” as a core dependent variable, or an explicit outcome of leadership. By using this 
model, content areas more appropriately reflect leadership in today's organizations. Therefore, organizational level 
(Rummler & Brache, 1995) performance outcomes of team management (Baker, Walsh, & Marjerison, 2000) and 
strategic leadership competencies (Collins, Lowe, & Arnett, 2000), which enable high-performance leaders to lead 
strategically in an environment of continuous change, were incorporated into the content areas. Baker et al. (2000) 
describe team management as the development and leadership of strong, effective organizational teams and the 
promotion of responsibility for team performance. Strategic leadership includes transforming culture and values of 
the organization, implementing change, and promoting continuous organizational improvement (Collins et al., 
2000 ). 

Results and Discussion 

While management development literature from 1986 through 2000 was reviewed, all eighteen (18) studies located 
for this research occurred in the time span from 1991-1999. No management development studies with performance- 
level outcomes were found in the literature from 1986-1990. 

Methods of Evaluation 

Quantitative or Qualitative Method. The measurement of organizational performance often involves both 
quantitative and qualitative testing at both the learning and the performance level of the Results Assessment System. 
Both qualitative and quantitative methods are important in analyzing strategic leadership interventions, including 
how strategic leadership can be developed and used to create cultures in which both people and performance are 
valued. The ideal assessment for organizational performance incorporates both quantitative and qualitative 
measurement methods. 

Five studies (30%) in this research exhibit a mixture of quantitative and quantitative assessment methods 
(Koene, Pennings, & Schreuder, 1992; Murphy & Settich, 1992; Riechmann, 1992; Sashkin, Rosenbach, Deal, & 
Petersen, 1992; Watad & Ospina, 1999). However, the most common method of assessment in this research was 
quantitative (n = 12, 66%). But, according to Schein (1990), no quantitative method alone can assess culture. 
Therefore, changes in performance based upon transformational leadership should be evaluated with both qualitative 
and quantitative methods to produce the best evaluation results. 

Six percent, one study (Riechmann, 1992), used qualitative research methods. Through qualitative research 
we can learn how to help leaders develop and create or change cultures. Qualitative research can help define the 
important variables and constructs and determine what is worth measuring, at which point it becomes appropriate to 
turn to quantitative methods of measurement. 

Measurement of Informal Training. One study (Uhl-Bien & Graen, 1992) utilized developmental 
relationships, or strong leader-follower relationships, as the management development intervention. While there are 
more coaching, action learning, self-development processes, mentoring, and peer-related learning activities 
(Garavan, Bamicle, & O’Suilleabhain, 1999; McCauley et al., 1998) occurring in organizations than fifteen years 
ago, these informal training methods are more difficult to measure as they cannot be viewed in an objective manner. 

Pretest-Posttest Measurements. Four studies (22%) included in this research utilized pretest-posttest 
measurements, or random assignment to experimental and control groups parling, Weber, & Kelloway, 1996; 
Moxnes & Eilertsen, 1991; Sashkin et al., 1992; Svyantek & DeShon, 1992). This result is not surprising, as 
research reported in management development literature most often does not typically have control groups or 
random assignment processes. 
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Performance-Level Evaluation 

Of the studies reviewed, four (22%) intentionally evaluated the management development experience at 
both the performance and learning levels (Barling et al., 1992; Howell & Avolio, 1993; Koene et al., 1992; Sashkin 
et al., 1992). All other studies (n = 14, 78%) completed management development evaluations on performance 
level data only. Of performance-level evaluations, nine studies (64%) evaluated system performance only, four 
(29%) evaluated financial performance only, and one (7%) evaluated both system and financial performance (Koene 
et al., 1992). 

An observation from this research is that systems can vary within an organization and so can the outcomes. 
Six studies (33%) measured performance of business units, teams, or groups, which are considered systems within 
an organization (Avolio & Howell, 1992; Howell and Avolio, 1993; Penwell, 1992; Riechmann, 1992; Spoth, 1992; 
Uhl-Bien & Graen, 1992). However, research indicates that there is a problem in linking leadership, in terms of 
individual behavior and characteristics to a performance-level variable as organizational culture (Sashkin et al., 
1992). 

Timing of Evaluation 

While most management development evaluations are at the learning-level and measure trainees’ reactions 
at the end of the program, analysis of the 18 studies in this research indicates that performance-level measurement of 
management development programs requires a greater commitment of time than evaluation at learning-levels 
(Swanson and Holton, 1999). The evaluation process took two or more years in fifty percent of the studies (n=9). 
Svyantek and DeShon (1992) evaluated the impact of the intervention over a 17-year time span. The evaluative 
study by Spoth (1992) covers a 14-year time span. It is interesting to note that these two studies involve large 
corporations, one being Chrysler and the other Fortune 500 companies. Three studies required five years for the 
completion of the evaluation (Fullagar, 1992; Williams, Greene, & Bergman, 1992), one required four years 
(Sashkin et al., 1992), and four studies covered a time span of three years (Moxnes & Eilertsen, 1991; Uhl-Bien & 
Graen, 1992; Watad & Ospina, 1999; Westcott, 1995). Four management development studies in this research 
required a time span of one to two years (Avolio & Howell, 1992; Barling et al., 1996; Glynn & Slepian, 1992; 
Howell & Avolio, 1993). Some studies did not record a time frame for the evaluation process (Koene et al., 1992; 
Lohmann, 1992; Murphy & Settich, 1992; Penwell, 1992; Riechmann, 1992). 

Because organizational changes often take many years, the evaluation of organizational performance 
outcomes also takes a long-term commitment for HRD. Organizations cannot for the most part make a long-term 
time commitment for evaluation of management development activities. Career changes by HRD professionals, top 
management, or trainees often slow down the evaluation process or causes the results not to be reported. 

In today’s environment, work is fluid, organizations are flatter, organizational structures frequently change, 
and people make more lateral movements (Byham, 1999). In addition, most training likely occurs at the grass roots 
level, leading to more evaluations at the individual than organizational level (Rummler & Brache, 1995). Also, 
managers who are responsible for evaluation are less likely to evaluate interventions affecting their own 
performance, as they perhaps would be fearful of the end result since it applies directly to them. 

Instrumentation 

Both quantitative and qualitative methods were used to evaluate performance-level outcomes in fourteen 
studies (78%). Four studies, or 22%, used one instrument to evaluate effectiveness (Glynn & Slepian, 1992; 
Lohmann, 1992; Riechmann, 1992; Spoth, 1992). 

Quantitative methods were found in 17 (94%) studies. Those studies used instruments developed outside 
of the organization, such as Argyris Learning Model, Repertory Grid Technique, Hofstede construct, Rotter’s locus 
of control, Jackson’s personality inventory, Multifactor Leadership Questionnaire (MLQ), Bass’ leader charisma 
scale, Leadership Behavior Questionnaire (LBQ), Leadership Description Questionnaire (LDQ), and Project Leader- 
member Exchange Scale (PMLX). Quantitative methods measuring group performance often used the SYMLOG 
(Systematic Multiple Observation of Groups) instrument. The only qualitative study in this research (Riechmann, 
1992) used self-reports as the instrumentation method. 
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Intervention Categories 

Job assignments (JA) were the most common interventions in this analysis (n = 9, 50%). Examples of job 
assignment experiences were transformational leadership changes in ideology, implementation of shared values, or 
succession of presidents. Formal training programs (FT) were found in five studies (27%) with a strategic emphasis 
on executive leadership as the primary focus. Of the remaining studies, three studies (17%) incorporated structured 
experiences (Penwell, 1992; Riechmann, 1992; Westcott, 1995), and one study (5%) had developmental 
relationships as the intervention (Uhl-Bien & Graen, 1992). 

Transformational leadership, encompassed in this study in the job assignment (JA) intervention category, 
functionally changes the strategic direction at the organizational level. According to Friedman (2000), 
“management always matters, but in this more complex and fast-paced system, management and strategic vision 
matters a lot more” (p. 231). Therefore, it is important to align the goals for management development programs 
with the strategic vision of the organization, and to train managers in their new roles in strategic management. 



Twelve studies (66%) focused on strategic leadership as the primary the content area for interventions. This 
is not surprising as the occurrence of strategic leadership interventions can be attributed to the need of organizations 
to change to meet the demands of a competitive environment as a result of globalization. 

Recommendations for Future Research 

HRD must make huge strides in the development of performance-level evaluation activities (Swanson & Holton, 
1999), especially those that test long-term impact of the intervention on the organization. Some topics that should be 
researched further are described in the following paragraphs. 

HRD professionals must take the lead in addressing the lack of performance-level evaluation methods. A 
limited number of management development programs with organizational outcomes are reported in the literature. 
Obviously, standard evaluation methods are not enough for organizations to measure organizational-level 
performance improvement (Rummler & Brache, 1995). Because organizations are facing a multitude of outcome- 
based demands on their time and resources, the development of an evaluation instrument should not be left up to the 
management of the organization. HRD should take the lead in combining evaluation theory with performance-based 
management development theory to create the appropriate system for measurement of organizational-level 
performance improvement. Many organizations utilize individual learning outcomes to reflect performance at the 
organizational level. However, there is no research that explicitly justifies that learning at the individual level 
translates to organizational performance. Further research must be done on the outcomes of management 
development programs at the organizational level (Rummler & Brache, 1995). 

HRD must take the initiative to link management development programs to organizational strategy. 
Management development programs have received criticism in recent years. Sometimes management training 
programs fail because they have no connection to real life in the company (Berry, 1990; Carlisle & Henrie, 1993) or 
fail to add value to corporate strategy. Traditionally training and development systems are relegated to narrowly 
defined support roles, where individuals are trained around current job-based deficiencies or predicted knowledge 
and skill needs (Olian, Durham, Kristoff, Brown, Pierce, & Kunder, 1998). For results to occur, the intervention 
must be linked with organizational goals and have utility or payoff to the organization. HRD should take the lead by 
strategically aligning training and development systems that advance and sustain the organization’s competitive 
position in its market. 

HRD must become the learning centers for global organizations. An inconsistency occurs when HR 
practitioners, operating in the same knowledge-based paradigm, are required to enable radical changes in 
organizations in response to rapid changes in business environments (Beddowes, 1994). HRD practitioners must 
take the lead with management development interventions in global organizations. However many HRD 
practitioners must change the basic way of doing their business, increase their knowledge about globalization, and 
change their perspectives to function in global organizations. HRD should be the learning centers for global 
organizations. As a result, HRD professionals must change from the Kirkpatrick (1996) reaction paradigm of 
evaluation to one that measures effectiveness of strategic development initiatives. Practitioners who use the four- 
level approach alone are quite likely to arrive at erroneous conclusions about their training programs in global 
organizations. Therefore, it is the responsibility of HRD to create and use a model that has the appropriate 
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constructs, which can be used as a diagnostic tool for determining critical influences that need to be measured along 
with global outcomes. 

Conclusion 

The evaluation task is daunting. Research reported in the literature on the effectiveness of management 
development programs is sparse, partially because of ineffective performance-level evaluation methods. However, 
high-performance leadership organizations always assess the impact of their management development processes 
(Collins et al., 2000; Fulmer & Wagner, 1999) and use that gap analysis to make improvements. Program 
evaluations provide the data to generate buy-in and to focus on the current mission-related objectives. 

HRD professionals should develop performance-level evaluation methods so that organizations develop the 
bench strength to compete globally. However, the challenge is huge for HRD because the type and intensity of the 
evaluation process depends on the objectives of the management development effort and the organization’s culture. 
Therefore, evaluation methods must be specific, but yet broad enough, to satisfy the evaluation needs of all 
organizations. Meanwhile, refined evaluation methods are needed to conduct more empirical research in the future 
on performance-level outcomes of management development programs. 
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Redefining Performance: Productivity and Return on Investment in Physical Therapy 
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In the field of physical therapy, the measurement of performance has been predominantly 
accomplished through productivity calculations . Return on investment (ROl) has been shown as 
an effective tool for evaluating performance in other businesses. Without a practical model for 
determining the ROl in this service-oriented healthcare field, managers must create useful tools to 
accurately measure human performance. This theoretical paper begins the conceptual analysis of 
such an instrument to measure ROl in physical therapy practice. 

Keywords: Return on Investment, Productivity, Performance 

Recent shifts in healthcare reimbursement have caused managers and employees to redefine their understanding of 
performance. Many workers are experiencing longer shifts and increased workload to meet the job demands of their 
company. Using Swanson’s (1999) definition of performance as the “valued productive output of a system in the 
form of goods and services” (p. 5), traditional measures for evaluating the actions of employees and programs in 
healthcare may no longer be appropriate. With the diverse dynamics of the modem workforce, each profession must 
discover methods to appropriately measure performance by building models or tools that focus on its unique role 
(Holton, 1999). Productivity measures have been used extensively in healthcare to measure the performance of 
individuals, and yet, this concept has been poorly defined. Productivity may, as Kaplan and Norton (1996) suggest, 
inadequately reflect the accomplishments of the employee in relation to the desired outcomes. This calculation may 
be inaccurate, misleading, and unrealistic in today’s healthcare environment. 

The profession of physical therapy (PT) is a healthcare sector, with particularly heavy reliance on 
productivity as a performance measure. Productivity has been widely used in PT practice to measure the 
performance and potential revenue from an individual, group, and/or a clinical program (e.g. Bohannon, 1987, 1984; 
Dupont, Gauthier-Gagnon, Roy, & Lamoureux, 1997), and to determine the compensation and rewards for clinicians 
and therapy departments (Nosse, Friberg, & Kovacek, 1999). However, few studies have related the concept of 
productivity to the financial results and measures of human performance. 

The American Physical Therapy Association (APTA) has developed the Guide to Physical Therapist 
Practice, which serves as a tool to outline and explain the abilities, capabilities, and skills of the members of the PT 
profession (1999). In this document, the performance of a physical therapist is included in five distinct roles: 
administration, education, critical inquiry (research), consultation, and patient/client management (practitioner). 
These roles are new to many therapists and expand the career opportunities and educational requirements of the 
field. These roles become evident through the documentation of outcomes for administration, education of clients, 
problem-solving approaches from research, consultation with other practitioners, and management of client 
caseloads. However, a model does not currently exist to effectively evaluate the employee’s performance within 
these roles. 

One such instrument for determining the outcomes and effectiveness of performance and programs is return 
on investment (ROl). ROl is often used in the field of human resource development (HRD) to measure the outcome 
of a program or training. ROl has been defined as the culmination of all activities of a company (Rachlin, 1997), 
and as both a management and a financial tool for measuring performance and company investments (Friedlob & 
Plewa, 1996). ROl has also been shown as a valid tool for measuring the performance of employees (Fitz-enz, 
2000). By calculating the ROl of performance, PT managers or supervisors can determine both the benefits and 
costs of their employees and of the company’s clinical programs. 

Research Problem 

This paper is intended to assist PT managers in identifying areas of opportunity for individual, team, and 
department/program growth within the five roles of the physical therapist. With an emphasis to increase employee 
productivity in the clinical setting, a need exists in the PT profession to relate productivity to performance, and to 
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identify factors, other than productivity, that can be investigated to measure ROI. The measurement of performance 
in terms of ROI may yield different results than the traditional calculation of productivity. If therapists demonstrate 
the ability to perform the five essential roles, is productivity the most valuable indicator of performance and 
potential revenue? The purpose of this paper is to begin the conceptual analysis of a model that may lead to a useful 
instrument, which evaluates performance in PT clinical practice. This paper investigates productivity and ROI 
models from multi-disciplinary sources and attempts to link related concepts and methods from the business and 
management field to the practice of PT. 

Theoretical Framework 

The theoretical framework for this paper concerns the proposal of a conceptual analysis for the development of a 
model that serves to measure the return on investment of a healthcare delivery system. 

HRD and Measuring Performance 

Performance has often been measured by conducting performance appraisals and comparing performance 
to benchmarks (DeSimone & Harris, 1998). Determining an all-encompassing definition of performance in the 
workplace has been a difficult task for HRD. Performance may be measured in terms of individual, process, and 
organizational perspectives (Rummler and Brache, 1992), or in relation to quantity, time, and quality (Swanson, 
1999). The relationship between HRD and performance is based in part on their focus of measuring output in order 
to identify areas for improvement. The measurement of performance allows the organization to evaluate the 
efficiency and the effectiveness of its efforts (Bates, 1 999). 

ROI represents a value that compares program costs to benefits and has been used as a benchmark for 
increasing profits, establishing objectives, and measuring results in businesses worldwide (Rachlin, 1997). It has 
been used extensively to reduce the factors of intuition and judgment and to measure the performance of 
management and employees (Friedlob & Plewa, 1996). In HRD, Phillips (1996) discussed the role of ROI in 
training evaluation to measure the monetary benefits in relation to the training costs. Phillips (1998) later 
emphasized the need to test and measure performance to ensure acquisition of skill and knowledge. By shifting the 
ROI model from human resources and management to human performance, managers may more precisely assess the 
knowledge, skills, and abilities of their employees. 

Background 

Characteristics of Physical Therapy 

PT involves the preservation, development, and restoration of physical function (APTA, 1999). Single 
therapy treatments may last from 15 minutes to over 2 hours per patient. The billing standards also vary depending 
on the payer for the service. In some cases, there are specific charges for given treatments. For example, if a client 
requires a computerized strength test, there may be a set fee for this action. PT may also be billed in time 
increments. Typically, therapy is billed to a third party payer in units. One unit is equivalent to 15 minutes of 
therapy. In 1997, the Healthcare Financing Administration implemented the Balanced Budget Act, which imposed 
the Prospective Payment System (PPS), which modified the billing system and set dollar amounts for certain 
treatments and group activities (Department of Health an Human Services, 1998). PT departments have 
consequently undergone changes in their methods of reimbursement. As managers seek to increase reimbursement, 
clinicians may be requested to increase their levels of productivity. These changes in reimbursement, as well as 
staffing changes in the clinic, indicate that productivity may not be the most accurate indicator of performance and 
positive financial results. 

PPS requires that therapists calculate the number of minutes of therapy for a patient on a weekly basis to 
determine the dollar reimbursement. To calculate an individual’s productivity, the clinician would divide the 
number of minutes billed during the day by the number of minutes he or she was able to provide treatment. 
Clinicians may choose to treat clients in groups of two or more depending upon the clinical program and the 
insurance stipulations. Clinics may require the therapist to average a predetermined level of productivity in order to 
remain in good standing with the company or to receive bonuses and promotions. This formula does not, however, 
account for the changes in reimbursement structure or the actual revenue or outcomes from PT services. 

Productivity is commonly used to measure performance in PT for many reasons. It is a relatively quick and 
simple figure to calculate, as opposed to the concepts of service and quality, or “soft” data as described by Phillips 
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(1998), and it provides a percentage to establish benchmarks or compare employees. The debate concerning the 
effectiveness of measuring productivity may stem from the definition and usage of the term itself. 

Productivity has been defined as the number of patients treated divided by the number of treatments 
(Dupont et al., 1997), and as the amount of time spent conducting patient care (Ladyshewsky, 1995; Ladyshewsky, 
Bird, & Finney, 1994). However, these examples only deliver a partial view of human performance because only 
one variable is considered. Investigating other possible factors that influence productivity would offer a more 
comprehensive measure of performance (Risher & Fay, 1995). Nosse et al. (2000) have defined productivity as the 
relationship between the values of the resources used —or input— and the value of what is produced —or output. The 
authors were identifying organizational processes, but this definition parallels Swanson’s concept of performance 
(1999) by investigating values in terms of dollars, outcomes, and company growth, and initiating therapy managers 
to expand their definition for productivity. 

Potential of ROI for Measuring Performance in PT 

Numerous formulas for ROI in healthcare have been proposed (e.g. Japsen, 1997; Rosenstein, 1999). 
However, these models measure the use of equipment and clinical decisions and not human performance factors. 
Although ROI models have not been proposed in the PT clinical setting to measure individual, group, and program 
performance, a financial model for clinical education programs has been presented (Lopopolo, 1984). Lopopolo’s 
model provides useful insight into the development of a framework for discussing ROI and its relation to 
productivity by measuring the costs for operating a department in terms of capital and business expenses, rather than 
human performance. Ladyshewsky and Barrie (1996) have also developed a tool to audit costs and benefits in 
clinical education. Components of this model may be useful in developing ROI when calculating program costs. 
Although group performance and supervision has not been studied in the clinical setting, group productivity in the 
clinical education of students has been shown to increase with appropriate supervision (Dupont et al, 1997). The 
relationship between productivity and ROI may provide relevant insight to collaborative models being implemented 
in clinical practice (Zavadak, Dolnack, Polich, & Van Volkenburg, 1995). 

The identification of other factors than productivity in developing a model is critical to measuring the 
performance of the five roles. One should be able to evaluate the learned skills of administration and education, the 
manual and “hands-on” skills of the clinician, and the interpretive and research abilities of the consultant and critical 
inquirer. Traditionally, PT has focused ROI on the administrative function, even though therapists are trained to 
perform in all five roles. To conceptually analyze performance at these five domains, an understanding of the 
present method of measurement is needed. Then, one can begin to develop a new approach to calculating ROI in the 
PT profession. 

Exploring a ROI Model for PT 

The determination of ROI for this paper consists of the calculation of direct and indirect costs, development costs, 
overhead costs, and compensation and benefits for PT staff, and for calculating the revenue generated in clinical 
practice. Since healthcare often gains economic benefits from cost reduction, instead of increasing revenue, therapy 
managers may be able to increase ROI by finding ways to decrease the costs to the department without any change 
in productivity. Managers need to be aware of these potential cost-saving measures in order to adequately evaluate 
the performance of the staff. 

As in business and HRD, the essential factors that directly affect the ROI for the clinic include productivity, 
promotability, transferability, and retainability. These factors have been shown to highly correlate with increased 
ROI (Fitz-enz, 2000; Rachlin, 1997). The factor of promotability captures the potential of an employee to advance in 
his or her job role. This advancement may be a part of an individual’s career development (Greenhaus, 1987). 
Transferability is the ability of the employee to learn new or different job skills, or improve his or her knowledge 
base. This component may be evaluated by managers to better understand the learning component of performance 
for the employee. Learning is associated with a “change” in the behavior, cognition, or affect of the employee that is 
a result of acquiring a new skill or knowledge (DeSimone & Harris, 1998). Retainability refers to the organization’s 
ability to retain an employee’s services. Employee retention may save an organization time and money by reducing 
training costs, decreasing the hiring of temporary workers, and increasing job satisfaction ratings (Fitz-enz, 2000). 
The extent to which individuals, groups, and/or programs have an effect on ROI depends upon the appropriate 
evaluation technique as well as appropriate self-assessment data for these factors. 

A traditional belief system in the PT profession is that increased productivity will lead to promotion, 
retention, and job transfer (Bohanon 1987, 1984; Domenech et al., 1983). By meeting or exceeding expectations in 
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productivity, one can move from the patient/client manager role to administration or consulting. Increased 
productivity may also warrant job recognition, which may lead to the transfer to other roles. If productivity is linked 
to compensation and benefits, employee retention may also be improved. This traditional system emphasizes one 
determinant--productivity--and its role in performance (see Figure 1). If productivity is not the most accurate 
indicator of performance, and ROl involves more measures than productivity, the use of productivity as the sole 
indicator of performance may be misleading. 



^^Troductivit^^^ 




Promotability 



Retainability 




Transferability 



Figure 1. Traditional Belief System for Productivity. Productivity expectations influence job promotions, 
employee retention, and the potential transfer of knowledge and careers. 

Findings and Results 

A link between ROl and productivity in physical therapy may be made by incorporating productivity, transferability, 
and retainability equally into the ROl model as shown in Figure 2. These three essential factors are measured to 
evaluate performance and ultimately, the opportunity for promotion to other roles of the physical therapist. If ROl is 
viewed by employees and managers as a more accurate measure of performance, then these related concepts could 
positively influence productivity, improve the transfer of learning, and retain valuable employees. For example, if 
one can relate increased production with retention or the opportunity for job transfer, then he or she may increase the 
probability of job promotion. By calculating the productivity measurements for the individual, group, and program, 
accurate comparisons among clinicians can be made. A manager can divide the anticipated revenue by the sum of 
the compensation, benefits, training, marketing, and program development costs to yield a more accurate view of 
ROl for the practice than the traditional measure of productivity alone. 

PT managers should be able to adequately evaluate these ROl factors in order to lead to an effective model. 
A therapist’s ability to transfer to different therapy environments involves the acquisition of new skills and/or 
knowledge. This attainment may be measured through the employee’s attendance in continuing education courses or 
the development of research for the profession. Transfer may include the placement of the employee in a different 
setting based upon learned abilities or skills. Learning opportunities can also be obtained through continuing 
education or on-the-job training and may add to the existing proficiencies of the employee. The effects of retention 
may be measured through leadership, responsibility, and management skills, as well as time spent with the 
organization. Productivity involves the measurement of manual skills as well as time management abilities. By 
appropriately assessing each factor, one may determine which one of the five roles best describes the individual. 
Table 1 identifies potential relationships between the five essential roles of the therapist with examples of possible 
measurement tools for each ROl factor. 
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Table 1 . Measurement of ROl in PT Practice in Relation to Job Roles 



Factors 


Measurement 


Role (examples) 


Transferability (learning) 




Continuing education 


Education/ 
Pt/Client Mgmt. 




Research 


Critical Inquiry 




Program development 


Consultant/Admin 




Knowledge and skill acquisition 


Pt/Client Mgmt./ 
Education 


Retainability (development) 




Management 


Administration/ 
Pt/Client Mgmt. 




Leadership 


Admin/Consultant 




Time in service to organization 


Pt/Client Mgmt./ 
Administration 


Productivity (skill) 




Manual skills (psychomotor) 


Education/Consultant/ 
Pt/Client Mgmt. 



Conclusions 

Healthcare practices are often unaware of the potential opportunities to improve and measure human performance in 
the workplace. Physical therapists were once trained to become educators of the patients, managers of their 
caseloads, and clinicians that are efficient in their clinical skills. Now, the roles have expanded and the job 
requirements have increased. Even though these domains of the therapist may be diverse, only the efficiency of the 
practice is routinely measured. This concept of performance remains new and difficult to explain for many 
professionals (Swanson, 1999). 




Transferability 




Retainability 


plijPi 


Productivity 


Leaming/Cognitive 




Development 


Hlr 1 


Skill 




Figure 2. Model for ROl in PT Practice. Promotion is dependent upon transfer of learning, job retention, 
and productivity. Dividing by the costs associated with human performance would yield the return on 
investment in the physical therapy clinical environment. 



Accurately measuring the R01 of human performance may be one of the most difficult tasks facing the PT 
profession today. The role of the HRD professional has been stated to solve individual and organizational problems 
by enhancing performance in multiple dimensions and demonstrating value to the learner (Ruona, 2000). Watkins 
(2000) stated that HRD is about making a difference in people’s lives and being a helping profession. Similarly, PT 
is founded on a belief of providing help to others in order to improve their physical performance. By designing a 
tool to assist PT managers to accurately evaluate the performance of their employees, HRD professionals can assist 
in fostering this meaningful opportunity for learning and change. 

Contribution to New Knowledge in HRD 

As healthcare continues to evolve and the financial reimbursement changes, new methods for managing patients 
need to be identified. While many of the administrative functions of a PT clinic undergo major transitions, patients 
continue to require the high level of skill that physical therapists are trained to provide. Over the past few decades, 
therapists have been challenged by the need to increase productivity without a reduction in service to the patients. 
Giving extra time to one patient often means another patient may suffer from the lack of physical therapy 
intervention. Even though therapists have had to become jugglers of time and masters of scheduling, they are still 
measured by archaic formulas that only address one component of their multidimensional talents. By addressing 
attention to other dimensions, managers and administrators may gain insight to these human elements of 
performance and realize the potential growth of ROI in their clinical practices. 

Although this conceptual analysis is intended to explore productivity’s role in ROI and to provide a 
practical tool for therapists, it is not intended to provide an all-encompassing measurement device for the profession 
to use in all practice and educational settings. ROI, rather than productivity, may serve to provide a more accurate 
indicator of the performance of staff and programs in PT clinical departments. Therefore, future research is required 
to implement a model in the clinical setting to evaluate the performance of both the individual and organizational 
components of physical therapy. These findings can then be compared to prior productivity calculations to produce 
an accurate determinant of performance for the profession. 
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